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Abstract 

We investigate the role of Massey's directed information in portfolio theory, data compression, and statistics with 
causality constraints. In particular, we show that directed information is an upper bound on the increment in growth 
rates of optimal portfolios in a stock market due to causal side information. This upper bound is tight for gambling in a 
horse race, which is an extreme case of stock markets. Directed information also characterizes the value of causal side 
information in instantaneous compression and quantifies the benefit of causal inference in joint compression of two 
stochastic processes. In hypothesis testing, directed information evaluates the best error exponent for testing whether a 
random process Y causally influences another process X or not. These results give a natural interpretation of directed 
information I(Y n — » X n ) as the amount of information that a random sequence Y n — (Yi, Y2, . . . , Y n ) causally 
provides about another random sequence X n = (Xi,X2, ■ ■ ■ ,X n ). A new measure, directed lautum information, is 
also introduced and interpreted in portfolio theory, data compression, and hypothesis testing. 

Index Terms 

Causal conditioning, causal side information, directed information, hypothesis testing, instantaneous compression, 
Kelly gambling, Lautum information, portfolio theory. 

I. Introduction 

Mutual information I(X; Y) between two random variables X and F arises as the canonical answer to a variety 
of questions in science and engineering. Most notably, Shannon [1] showed that the capacity C, the maximal data 
rate for reliable communication, of a discrete memoryless channel p(y\x) with input X and output Y is given by 

C = maxI(X;Y). (1) 

p{x) 

Shannon's channel coding theorem leads naturally to the operational interpretation of mutual information I(X; Y) = 
H(X) — H(X\Y) as the amount of uncertainty about X that can be reduced by observation Y, or equivalently, 
the amount of information that Y can provide about X. Indeed, mutual information I{X; Y) plays a central 
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role in Shannon's random coding argument, because the probability that independently drawn sequences X n = 
(Xi, X2, ■ ■ ■ , X n ) and Y n = (Yi, Y2, ■ . . ,Y n ) "look" as if they were drawn jointly decays exponentially with 
I(X;Y) in the first order of the exponent. Shannon also proved a dual result [2] that the rate distortion function 
R(D), the minimum compression rate to describe a source X by its reconstruction X within average distortion D, 
is given by R(D) = min p ^\x) In another duality result (the Lagrange duality this time) to (Q}, Gallager 

[3] proved the minimax redundancy theorem, connecting the redundancy of the universal lossless source code to 
the maximum mutual information (capacity) of the channel with conditional distribution that consists of the set of 
possible source distributions (cf. [4]). 

It has been shown that mutual information has also an important role in problems that are not necessarily related 
to describing sources or transferring information through channels. Perhaps the most lucrative of such examples 
is the use of mutual information in gambling. In 1956, Kelly [5] showed that if a horse race outcome can be 
represented as an independent and identically distributed (i.i.d.) random variable X, and the gambler has some side 
information Y relevant to the outcome of the race, then the mutual information I(X;Y) captures the difference 
between growth rates of the optimal gambler's wealth with and without side information Y. Thus, Kelly's result 
gives an interpretation of mutual information I(X; Y) as the financial value of side information Y for gambling in 
the horse race X. 

In order to tackle problems arising in information systems with causally dependent components, Massey [6] 
introduced the notion of directed information, defined as 

n 

I(X n — Y n ) —^/(X^YilY^ 1 ), (2) 
i=i 

and showed that the normalized maximum directed information upper bounds the capacity of channels with feedback. 
Subsequently, it was shown that Massey's directed information and its variants indeed characterize the capacity of 
feedback and two-way channels [7]— [13] and the rate distortion function with feedforward [14]. Note that directed 
information (01 can be rewritten as 

n 

I{X n — Y n ) =^Tl(X i ;Y- n \X i - 1 ,Y i - 1 ), (3) 

i—l 

each term of which corresponds to the achievable rate at time i given side information (X l ~ 1 , Y 1 ^ 1 ) (refer to [11] 
for the details). 

The main contribution of this paper is showing that directed information has a natural interpretation in portfolio 
theory, compression, and statistics when causality constraints exist. In stock market investment (Sec. UTTb . directed 
information between the stock price X and side information Y is an upper bound on the increase in growth rates 
due to causal side information. This upper bound is tight when specialized to gambling in horse races. In data 
compression (Sec. IPvl we show that directed information characterizes the value of causal side information in 
instantaneous compression, and it quantifies the role of causal inference in joint compression of two stochastic 
processes. In hypothesis testing (Sec.[V) we show that directed information is the exponent of the minimum type II 
error probability when one is to decide if Yi has a causal influence on Xi or not. Finally, we introduce the notion of 
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directed Lautumj information (Sec. lVH . which is a causal extension of the notion of Lautum information introduced 
by Palomar and Verdu [15]. We briefly discuss its role in horse race gambling, data compression, and hypothesis 
testing. 

II. Preliminaries: Directed Information and Causal Conditioning 

Throughout this paper, we use the causal conditioning notation (-||) developed by Kramer [7]. We denote by 
p(x n \\y n ~ d ) the probability mass function of X n = (X\, . . . , X n ) causally conditioned on Y n ~ d for some integer 
d > 0, which is defined as 

n 

p(x n \\y n - d ) :=l[p(x i \x i - 1 ,y i - d ). (4) 

i=i 

(By convention, if i < d, then x l ~ d is set to null.) In particular, we use extensively the cases d = 0, 1: 

n 

P(x n \\y n ):=Y[p(x i \x i - 1 ,y i ), (5) 
i=l 
n 

p^wv*- 1 ) -n^i^ 1 ,^ -1 )- (6) 

Using the chain rule, we can easily verify that 

p{x n 1 y n )=p{x n \\y n )p{y n \\x n - 1 ). (7) 
The causally conditional entropy H{X n \\Y n ) and H(X n \\Y n ~ 1 ) are defined respectively as 

n 

H(X n \\Y n ) :=E[\ogp{X n \\Y n )]=Y J H{X l \X l -\Y l ), 

i=l 

n 

H{X n \\Y n ~' 1 ) := E[\ogp(X n \\Y n - 1 )} =^2H(X i \X i ~ 1 ,Y i ~ 1 ). (8) 

i=l 

Under this notation, directed information defined in (O can be rewritten as 

I(Y n -► X n ) = H{X n ) - H(X n \\Y n ), (9) 

which hints, in a rough analogy to mutual information, a possible interpretation of directed information I(Y n — » X n ) 
as the amount of information that causally available side information Y n can provide about X n . 

Note that the channel capacity results [6]— [13] involve the term I(X n — > Y n ), which measures the amount of 
information transfer over the forward link from X n to Y n . In gambling, however, the increase in growth rate is 
due to side information (the backward link), and therefore the expression I(Y n — > X") appears. Throughout the 
paper we also use the notation I(Y n ^ 1 — * X n ) which denotes the directed information from the vector (0,y™ _1 ), 
i.e., the null symbol followed by Y 71 ^ 1 , to the vector to X n , that is, 

n 

Xyn-i _>x n ) =J2l(Y i - 1 ;X i \X i - 1 ). 

i=2 

Lautum ("elegant" in Latin) is the reverse spelling of "mutual" as aptly coined in [15]. 
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Using the causal conditioning notation, given in ©, the directed information I(Y n 1 — > X n ) can be written as 

j{Y n - 1 -+ X n ) = H{X n ) - H{X n \\Y n ~ x ). (10) 
Directed information (in both directions) and mutual information obey the following conservation law 

I{X n - Y n ) = I(X n -» Y n ) + I{Y n ~ l -4X"), (11) 

which was shown by Massey and Massey [16]. The conservation law is a direct consequence of the chain rule (£7), 
and we show later in Sec. IIV-BI that it has a natural interpretation as a conservation of a mismatch cost in data 
compression. 

The causally conditional entropy rate of a random process X given another random process Y and the directed 
information rate from X to Y are defined respectively as 

H{X\\Y):= lim g ( A ""H y ") | (12 ) 

n — >oo TL 

I(X n — > Y n ) 

1(X -> Y) := lim — '-, (13) 

n— >oo 71 

when these limits exist. In particular, when (X, Y) is stationary ergodic, both quantities are well-defined, namely, 
the limits in ( fT2] i and ( TT31 exist [7, Properties 3.5 and 3.6]. 

III. Portfolio theory 

Here we show that directed information is an upper bound on the increment in growth rates of optimal portfolios 
in a stock market due to causal side information. We start by considering a special case where the market is a horse 
race gambling and show that the upper bound is tight. Then we consider a general stock market investment. 



A. Horse Race Gambling with Causal Side Information 

Assume that there are m racing horses and let Xj denote the winning horse at time i, i.e., Xj G X := 
{1, 2, . . . , to}. At time i, the gambler has some side information which we denote as Yj,. We assume that the 
gambler invests all his/her capital in the horse race as a function of the previous horse race outcomes X 1 ^ 1 and 
side information Y l up to time i. Let 6(cc^ 1 , y l ) be the portion of wealth that the gambler bets on horse 
Xi given X^ 1 — x 1 ^ 1 and Y % — y l . Obviously, the gambling scheme should satisfy b(xi\x t ~ 1 , y l ) > and 
Tlx ex bixilx 1-1 , y l ) = 1 for any history {x 1 " 1 ,y % ). Let o(xi\x 1 ^ 1 ) denote the odds of a horse x.- L given the 
previous outcomes x 1 ^ 1 , which is the amount of capital that the gambler gets for each unit capital that the gambler 
invested in the horse. We denote by S(x n \\y n ) the gambler's wealth after n races with outcomes x n and causal 
side information y n . Finally, iW / (X"||Y' n ) denotes the growth rate of wealth, where the growth W^(X n ||Y") is 
defined as the expectation over the logarithm (base 2) of the gambler wealth, i.e., 

W(X n \\Y n ) :=E[logS(X n \\Y n )}. (14) 



Without loss of generality, we assume that the gambler's initial wealth So is 1. We assume that at any time n 
the gambler invests all his/her capital and therefore we have 

S(X n \\Y n ) = biXnlX 11 - 1 ,Y n )o{X n \X n - 1 )S(X n - 1 \\Y n - 1 ). (15) 

This also implies that 

n 

S(X n \\Y n ) =Y[b(X i \X i - 1 ,Y i )o{X i \X i - 1 ). (16) 

The following theorem establishes the investment strategy for maximizing the growth. 

Theorem 1 (Optimal causal gambling): For any finite horizon n, the maximum growth is achieved when the 
gambler invests the money proportional to the causally conditional distribution of the horse race outcome, i.e., 

b(x i \x i - 1 ,y i )=p(x i \x i - 1 ,y i ) Mi e {1, —,n}, x i e X\ y { e y\ (17) 

and the maximum growth, denoted by W*(X™| \Y n ), is 

W*{X n \\Y n ) := max W(X n \\Y n ) = E[logo(X n )] - H(X n \\Y n ). (18) 

{b(x i \x'- 1 ,y t - 1 )}f =1 

Note that since {p(xi\x i , y l )}" =1 uniquely determines p(x n \\y n ), and since y')}"=i uniquely 

determines b(x n \\y n ), then ( TTTb is equivalent to 

b(x n \\y n )=p{x n \\y n ). (19) 

Proof: Consider 

W*{X n \\Y n )= max E[\ogb(X n \\Y n )o{X n )} 

b(x n \\y n ) 

= max E[\ogb(X n \\Y n )}+ E[logo(X n )} 

b(x n \\y n ) 

= -H{X n \\Y n )+E[logo(X n )}. (20) 

Here the last equality is achieved by choosing b(x n \\y n ) = p(x n \\y n ), and it is justified by the following upper 
bound: 



E[\ogb(X n \\Y n )} = E 



V ' 



\ogp{x n \\y n ) +log 



= -i/(x»||y")+ E p(^,y")iog 

x n ,y n 

< -H(x n \\Y n )+\o g e p(^,y n : 

(6) 



6(x n 


\y n ) 


p(x 11 


\y n ) 


b(x n \ 


y n ) 


p(x n \ 


\y n ) 


b{x n 


\y n ) 


p(x n 


\y n ) 



H{X n \\Y n )+\og E p{y n \\x n - x )b{x n \\y n ) 



{ ^ -H(X n \\Y n ). (21) 

where (a) follows from Jensen's inequality, (b) from the chain rule, and (c) from the fact that 
E^. V "P(2/ n |l^ 1 )^ ,l ||2/") = l- ■ 
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In case that the odds are fair, i.e., o(Xi\X l *) = 1/m, 

W*{X n \\Y n ) = nlogm - H(X n \\Y n ), (22) 

and thus the sum of the growth rate and the entropy of the horse race process conditioned causally on the side 
information is constant, and one can see a duality between H(X n \\Y n ) and W^*(X n ||Y Tt ). 
Let us define AW / (X™||F n ) as the increase in the growth due to causal side information, i.e., 

AW(X n \\Y n ) = W*(X n \\Y n ) - W*{X n ). (23) 

Corollary 1 (Increase in the growth rate): The increase in growth rate due to the causal side information 
sequence Y n for a horse race sequence X n is 

-AW{X n \\Y n ) = -I(Y n -» X n ). (24) 

n n 

As a special case, if the horse race outcome and side information are pairwise i.i.d., then the (normalized) directed 
information —I(Y n — » X n ) becomes the single letter mutual information I(X; Y), which coincides with Kelly's 
result [5]. Proof: From the definition of directed information (O and Theorem Q] we obtain 

W*{X n \\Y n ) - W*(X n ) = -H{X n \\Y n ) + H{X n ) = I(Y n -» X n ). 

■ 

Example 1 (Gambling in a Markov horse race process with causal side information): Consider the case in 
which two horses are racing, and the winning horse Xi behaves as a Markov process as shown in Fig. [T] A 
horse that won will win again with probability p and lose with probability 1 — p (0 < p < 1). At time zero, we 
assume that the two horses have equal probability of wining. The side information revealed to the gambler at time 
i is Yi, which is a noisy observation of the horse race outcome JQ. It has probability 1 — q of being equal to X^, 
and probability q of being different from Xj. In other words, Y^ = Xi + Vi mod 2, where Vi is a Bernoulli^) 
process. 



Horse 1 wins P Horse 2 wins 




Fig. 1. The setting of Example 1. The winning horse Xi is represented as a Markov process with two states. In state 1, horse number 1 wins, 
and in state 2, horse number 2 wins. The side information, Yi, is a noisy observation of the winning horse, Xi. 

For this example, the increase in growth rate due to side information AW :— -^AW(X n \\Y n ) is 

AW = h(p*q)-h(q), (25) 
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where h(x) := —x log x — (l — x) log(l — x) is the binary entropy function, and p*q = {X—p)q+{l — q)p denotes 
the parameter of a Bernoulli distribution that results from convolving two Bernoulli distributions with parameters 
p and q. 

The increase AW in the growth rate can be readily derived using the identity in Q as follows: 

1 i n 

-I(Y n -> X n ) = -yify,;!"!! 1 - 1 ^'- 1 ) 
n n • <L - ' 

^HfyilXoj-HCyiiXi), (26) 

where equality (a) is the identity from ||3), which can be easily verified by the chain rule for mutual information 
[11, eq. (9)], and (b) is due to the stationarity of the process. 

If the side information is known with some lookahead k > 0, meaning that at time i the gambler knows Y l+k , 
then the increase in growth rate is given by 

AW = lim -I(Y n+k -» X n ) 

n—>oo ft 

= H(Y k+1 \Y k ,X )-H(Y 1 \X 1 ), (27) 

where the last equality is due to the same arguments as (l26l l, If the entire side information sequence (Yj., Y2, . . .) 
is known to the gambler ahead of time, then since the sequence H(Yk+i \Y k ~ 1 , Xq) converges to the entropy rate 
of the process, we obtain mutual information [5] instead of directed information, i.e., 

AW = lim -I(Y n ;X n ) 

= lim — l , --H(Y 1 \X 1 ). (28) 

B. Investment in a Stock Market with Causal Side Information 

We use notation similar to the one in [17, ch. 16]. A stock market at time i is represented by a vector Xi = 
(Xii,Xi2, . . . ,Xj m ), where m is the number of stocks, and the price relative Xn~ is the ratio of the price of 
stock-fc at the end of day i to the price of stock-A; at the beginning of day i. Note that gambling in a horse race is 
an extreme case of stock market investment — for horse races, the price relatives are all zero except one. 

We assume that at time i there is side information Y 1 that is known to the investor. A portfolio is an allocation 
of wealth across the stocks. A nonanticipating or causal portfolio strategy with causal side information at time i 
is denoted as b(x z_1 ,?/ i ), and it satisfies Y^k=i y i ) — 1 and 6fe(x J_1 , y l ) > for all possible (x l ~ 1 ,y i ). 

We define S(x n \\y n ) to be the wealth at the end of day n for a stock sequence x" and causal side information y n . 
We have 

S(x n \\y n ) = (b*(x"-\y") -x„) Stx"- 1 ^"- 1 ), (29) 
where b* • x denotes inner product between the two (column) vectors b and x. The goal is to maximize the growth 

W(X n \\Y n ) := E{\ogS(X n \\Y n )}. (30) 
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The justifications for maximizing the growth rate is due to [18, Theorem 5] that such a portfolio strategy will 
exceed the wealth of any other strategy to the first order in the exponent for almost every sequence of outcomes 
from the stock market, namely, if S'*(X n ||F") is the wealth corresponding to the growth rate optimal return, then 

limsup - log ( fi^JliCll ) < a.s. (31) 



n ° \S*(X n \\Y n ) i 
Let us define 

W(X n \X n -\Y n ) := S[log(b*(X"- 1 ,y")X„)]. (32) 
From this definition follows the chain rule: 

n 

W{X n \\Y n ) = ^V7(X i |X i - 1 ,F i ), (33) 

i=i 

from which we obtain 



max W{X n \\Y n ) = ^ max W^X^Y*) 

{b(x»- 1 ,2/*)}?L 1 ^b(x ! \y*) 

n „ 

= J2 /(x*- 1 ,^) max ^(X^-W), (34) 

where /(x 4_1 ,y l ) denotes the probability density function of (x l_1 ,y l ). The maximization in (1341 is equivalent 
to the maximization of the growth rate for the memoryless case where the cumulative distribution function of the 
stock-vector X is P(X < x) = Pr(X^ < x|x l_1 , y l ) and the portfolio b = b(x l_1 , y l ) is a function of (x* -1 , y l ), 
i.e., 

maximize ^logfVX)^- 1 = x l -\Y l = if] 

m 

subject to 2_. bk — 1, 

i=i 

b k > 0,Vfc G [l,2,...,m]. (35) 

In order to upper bound the difference in growth rate due to causal side information we recall the following result 
which bounds the loss in growth rate incurred by optimizing the portfolio with respect to a wrong distribution g(x) 
rather than the true distribution /(x). 

Theorem 2 ( [19, Theorem 1]): Let /(x) be the probability density function of a stock vector X, i.e., X ~ /(x). 
Let hf be the growth rate portfolio corresponding to /(x), and let h g be the growth rate portfolio corresponding 
to another density <?(x). Then the increase in optimal growth rate AW by using hf instead of h g is bounded by 



AW = £pog(b}X)] - £pog(b*X)] < (36) 

-M, 

a( x ) 



where D(f\\g) := j f(x)\og^jp-dx denotes the Kullback-Leibler divergence between the probability density 



functions / and g. 

Using Theorem [2] we can upper bound the increase in growth rate due to causal side information by directed 
information as shown in the following theorem. 
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Theorem 3 (Upper bound on increase in growth rate): The increase in optimal growth rate for a stock market 
sequence X" due to side information Y n is upper bounded by 



W*(X n \\Y n ) - W*(X. n ) < I(Y n -» X"), 

where W*(X n \\Y n ) = max {b(X i-i,y*)} ?=1 W(X n \\Y n ) and W*(X n ) := max {b(X i-i )}?=i W{X n ). 
Proof: Consider 



W*(X n \\Y n ) - W*(X n ) 



E 



= 1 ■'x* 



max ^(Xilx*" 1 ,^) - max ^(X^x^ 1 ) 

h^x*- 1 ^) b i (x i - 1 ) 



r \ n 



/(x 4 |x*-\y*)log 



/(xilx*- 1 ,^ 



/(x;|x* 



E^ 



log 



/(x,|x^\y*) 



/(Xi|X*-i) 

n 

= ^ Mx.ix*- 1 ) - MXi|x i_1 , r) 

= I(Y n -► X n ), 

where the inequality (a) is due to Theorem [2] 

Note that the upper bound in Theorem [3] is tight for gambling in horse races (Corollary Q]). 



(37) 



(38) 



IV. Data Compression 

In this section we investigate the role of directed information in data compression and find two interpretations: 

1) directed information characterizes the value of causal side information in instantaneous compression, and 

2) it also quantifies the role of causal inference in joint compression of two stochastic processes. 

A. Instantaneous Lossless Compression with Causal Side Information 

Let Xi , Xi ... be a source and Yi , Y2 , ■ • ■ be side information about the source. The source is to be encoded 
losslessly by an instantaneous code with causally available side information, as depicted in Fig. [2] More formally, 
an instantaneous lossless source encoder with causal side information consists of a sequence of mappings \Mi\i>\ 
such that each M, : ^f'x / h {0, 1}* has the property that for every x 1 ' 1 and y\ M^(a; I_1 •, y % ) is an instantaneous 
(prefix) code. 

An instantaneous lossless source encoder with causal side information operates sequentially, emitting the concate- 
nated bit stream M\{Xi, Y\)M2{X 2 , Y 2 ) . . .. The defining property that Mj(a: •, y l ) is an instantaneous code for 
every x 1 ^ 1 and y % is a necessary and sufficient condition for the existence of a decoder that can losslessly recover x % 
based on y % and the bit stream M\(x\, y\)M2(x 2 1 y 2 ) ... as soon as it receives Mi(x±, yi)M2(x 2 , y 2 ) . . . Mi(x l , y l ) 
for all sequence pairs (x\,y\), (0:2,2/2)) ■ ■ and all i > 1. Let L(x n \\y n ) denote the length of the concatenated 



10 



Encoder 



Y> 



Mi{X\Y l ) 



Decoder 



X i {M\Y i 



Fig. 2. Instantaneous data compression with causal side information 



string Mi(xi, yi)M 2 (x 2 , y 2 ) . . . M n (x n ,y n ). Then the following result is due to Kraft's inequality and Huffman 
coding adapted to the case where causal side information is available. 

Theorem 4 (Lossless source coding with causal side information): Any instantaneous lossless source encoder 
with causal side information satisfies 

1 1 " 

-EL(X n \\Y n ) > - ViTpfilX 1 - 1 ,}^) Vn>l. (39) 
n n z — ' 

i=i 

There exists an instantaneous lossless source encoder with causal side information satisfying 

1 1 " 

-EL(X n \\Y n ) < - Vr, + H{X l \X l - 1 ,Y l ) Vi > 1, (40) 

i=l 

where = Y^x*- 1 ^* P{ x% ~ X i V % ) mm (l, max x , p(a;j|x l_1 , + 0.086). 

Proof: The lower bound follows from Kraft's inequality [17, Theorem 5.3.1] and the upper bound follows 
from Huffman coding on the conditional probability p(xi\x 1 ^ 1 , y l ). The redundancy term follows from Gallager's 
redundancy bound [20], min(l,Pi + 0.086), where Pi is the probability of the most likely source letter at time i, 
averaged over side information sequence [X % ,Y % ). ■ 

Since the Huffman code achieves the entropy rate for dyadic probability, it follows that if the conditional 
probability p(xi\x l ~ 1 , y 1 ^ 1 ) is dyadic, i.e., if each conditional probability equals to 2~ fc for some integer k, then 
(l39l can be achieved with equality. 

Theorem g] combined with the identity £™ =i i/pQlX*" 1 , y*) = H(X n ) - I(Y n -> X n ), implies that the 
compression rate saved in optimal sequential lossless compression due to the causal side information is upper 
bounded by ±I(Y n -> X n ) - 1, and lower bounded by ±I(Y n -> X n ) + 1. If all the probabilities are dyadic, 
then the compression rate saving is exactly the directed information rate -^I(Y n — > X n ). This saving should be 
compared to —I{X n ;Y n ), which is the saving in the absence of causality constraint. 

B. Cost of Mismatch in Data Compression 

Suppose we compress a pair of correlated sources {pQ, y )} jointly with an optimal lossless variable length code 
(such as the Huffman code), and we denote by E(L(X n , Y n )) the average length of the code. Assume further that 
y is generated randomly by a forward link p(yi\y t ~ 1 , x 1 ) as in a communication channel or a chemical reaction, 
and Xi is generated by a backward link p(xi\y l ~ 1 ,x' l ~ 1 ) such as in the case of an encoder or a controller with 
feedback. By the chain rule for causally conditional probabilities (O, any joint distribution can be modeled according 
to Fig. 
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Forward link 







p{Vi\x l ,y l l ) 






Backward link 










p(xi\x l ~ 1 , y l ~ l ) 



Yi 



Fig. 3. Compression of two correlated sources {-?Q,Yj}j>i. Since any joint distribution can be decomposed as p(x n ,y n ) = 
p(x n \\y n ~ 1 )p(y n \\x n ), each link embraces the existence of a forward or feedback channel (chemical reaction). We investigate the influence 
of the link knowledge on joint compression of Vi}j>i. 



Recall that the optimal variable-length lossless code, in which both links are taken into account, has the average 
length 

H(X n ,Y n ) < E(L(X n , Y n )) < H(X n ,Y n ) + 1. 

However, if the code is erroneously designed to be optimal for the case in which the forward link does not exist, 
namely, the code is designed for the joint distribution p(y n )p(x n \ \y n ^ 1 ), then the average code length (up to 1 bit) 
is 



x",y" 



= ^ log p(a: " ";fl - +H{X",Y") 

±-?,n P(y n )p(x n \\y n - 1) 

= 7(X n ^y") + fl'(X n ,F n ). 



(41) 



Hence the redundancy (the gap from the minimum average code length) is I(X n — > Y n ). Similarly, if the backward 
link is ignored, then the average code length (up to 1 bit) is 

E(L(X n ,Y n ))= P(*",y n )log ; * ; 

Jr^n p(y n \\x n )p(x n ) 

= £ P(»", V n ) log ( P }f:f} n , + H(X n , Y n ) 
JrJ n p(y n \\x n )p(x n ) 

= £ pOc", y") log + H(X n , Y n ) 



I(Y 



n-l 



p(x n ) 
X n ) + H(X n ,Y n ) 



(42) 



Hence the redundancy for this case is I(Y n 1 — > X"). If both links are ignored, the redundancy is simply the 
mutual information I(X n ;Y n ). This result quantifies the value of knowing causal influence between two processes 
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when designing the optimal joint compression. Note that the redundancy due to ignoring both links is the sum of 
the redundancies from ignoring each link. This recovers the conservation law ( fTTT l operationally. 

V. Directed Information and Statistics: Hypothesis Testing 

Consider a system with an input sequence (Xi,X2, ■ ■ ■ ,X n ) and output sequence (Yi, Y2, •• • > Y n ), where the 
input is generated by a stimulation mechanism or a controller, which observes the previous outputs, and the output 
may be generated either causally from the input according to {p(yi\y t ~ 1 , x l )}f =1 (the null hypothesis Hq) or 
independently from the input according to {p(yi|2/ I-1 )}™ = i (the alternative hypothesis Hi). For instance, this setting 
occurs in communication or biological systems, where we wish to test whether the observed system output Y n is 
in response to one's own stimulation input X n or to some other input that uses the same stimulation mechanism 
and therefore induces the same marginal distribution p(y n ). The stimulation mechanism p(x n \\y n ~ 1 ), the output 
generator p(y n \\x n ), and the sequences X n and Y n are assumed to be known. 



Hypothesis Hq: 



Hypothesis Hi 





1 


1 




i 


Controller 




Output generator 


p{xi\x l 1 ,y l *) 


Xi 




p(yi\x\y l : ) 


Yi 







Xi 

A 



Controller 



p(xi\x % 1 ,y l *) 



Xi 



Y, 



i-l 



Output generator 



p(yi\y 



Y 



Fig. 4. Hypothesis testing. Hq: The input sequence (X 1 , X2 ■ ■ ■ , X n ) causally influences the output sequence (Yi, Y2, . . . , Yn) through 
the causal conditioning distribution p(y"\\x n ). Hi: The output sequence (Yj , Y2, . . . , Y n ) was not generated by the input sequence 
(X \,X'2, ■ ■ ■ ,Xn), but by another input from the same stimulation mechanism p(x n \\y n ^ 1 ). 



An acceptance region A is the set of all sequences (x n ,y n ) for which we accept the null hypothesis Hq. The 
complement of A, denoted by A c , is the rejection region, namely, the set of all sequences (x n ,y n ) for which we 
reject the null hypothesis Hq and accept the alternative hypothesis H\. Let 



a := Pr(A c \H ), (3 := Pr(A\Hi) 



(43) 



denote the probabilities of type I error and type II error, respectively. 

The following theorem interprets the directed information rate X(X — * Y) as the best error exponent of /3 that 
can be achieved while a is less than some constant e > 0. 

Theorem 5 (Chernoff-Stein Lemma for the causal dependence test: Type II error): Let (X, Y) — {Xi,YA- c *L 1 
be a stationary and ergodic random process. Let A n C [X x y) n be an acceptance region, and let a n and (3 n 
be the corresponding probabilities of type I and type II errors j43l . For < e < |, let 

P [ u ] = , min [3 n . (44) 
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Then 



lim --log (3^ = Z(X->Y), 



(45) 



where the directed information rate is the one induced by the joint distribution from Hq, i.e., p(x n | \y n 1 )p{y n \\x n ). 

Theorem [5] is reminiscent of the achievability proof in the channel coding theorem. In the random coding 
achievability proof [17, ch 7.7] we check whether the output Y n is resulting from a message (or equivalently 
from an input sequence X n ) and we would like to have the error exponent which is, according to Theorem [5] 
I(X n — > Y n ) to be as large as possible so we can distinguish as many messages as possible. 

The proof of Theorem combines arguments from the Chernoff-Stein Lemma [17, Theorem 11.8.3] with the 
Shannon-McMillan-Breiman Theorem for directed information [14, Lemma 3.1], which implies that for a jointly 
stationary ergodic random process 

1 p(Y n \\X n ) 

- l0 § p( yn } - Z(X - Y) in probability. 
Proof: Achievability: Fix 8 > and let A n be 



x n ,y n : 



p(y n ) 



Y) 



< 5 



(46) 



By the AEP for directed information [14, Lemma 3.1] we have that Pr(A n \Ho) — > 1 in probability; hence there 
exists N(e) such that for all n > N(e), a n — Pi(A^ l \Ho) < e. Furthermore, 



n = Pv(A n \H 1 ) 

= p(^ n lly n - 1 >(y n ) 

x n .y n eA n 

< Yl p(x n \\y n ~ 1 )p(y n \\x n )2- n{I{x ^ Y) ~ s) 

x n ^yn£j\ n 

= 2 -n(i(x^Y)-5) ^ p(x n \\y n - 1 )p(y n \\x n ) 

® 2 - n{I( X^Y)-6) il _ an ^ 



(47) 



where inequality (a) follows from the definition of A n and (b) from the definition of a n . We conclude that 



lim sup - log/3„ < -T{X -»y) + i5, 

n— >oo 72 



(48) 



establishing the achievability since 6 > is arbitrary. 
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Converse: Let B n C(Ix y) n such that Vv(B c n \H ) < e < \. Consider 
Pr(S„|ffi) >Pr(A„ni? n |i/ 1 ) 

E P(^ll2/" _1 M2/") 

(i",j»)eA„ns„ 

> ]T ^"ll^ 1 )p(y"ll^ 1 )2-" (I(x ^ y)+5) 

(i",j")6A„nB„ 

= 2 -"( J ( x ^ y )+' 5 ) Pr(A„ n B n |if ) 

= 2 -"( J ( x ^ y )+ 5 )(l -Pr(A£ UB£|tf )) 

> 2-"( J ( x ^ y )+ 5 )(l - Pr(^|Ho) - Pr(B£|# )) (49) 
Since Yi(A c n \H ) -> and Pr(B^| J ff ) < e < §, we obtain 

lim inf - log/3„ > -{1(X — > Y) + S). (50) 
Finally, since (5 > is arbitrary, the proof of the converse is completed. ■ 

VI. Directed Lautum Information 
Recently, Palomar and Verdu [15] have defined the lautum information L(X n ;Y n ) as 

L(X";Y") := £ ^)^|^- (51) 

and showed that it has operational interpretations in statistics, compression, gambling, and portfolio theory, where 
the true distribution is p{x n )p(y n ) but mistakenly a joint distribution p(x n ,y n ) is assumed. As in the definition 
of directed information wherein the role of regular conditioning is replaced by causal conditioning, we define two 
types of directed lautum information. The first type 



and the second type 

L 2 (X"^F"):= ^p( s -|| tf »-i) p ( tf »)Iog-^L. (53) 

x ,y 

When p(x n ||i/™ _1 ) = p(x n ) (no feedback), the two definitions coincide. We will see in this section that the 
first type of directed lautum information has operational meanings in scenarios where the true distribution is 
p(x n )p(y n ) and, mistakenly, a joint distribution of the form p(x n )p(y n \\x n ) is assumed. Similarly, the second 
type of directed information occurs when the true distribution is p(x n \\y n ^ 1 )p(y n ), but a joint distribution of the 
form p(x n \\y n ^ 1 )p(y n \\x n ) is assumed. 

We have the following conservation law for the first-type directed lautum information: 

Lemma 1 (Conservation law for the first type of directed latum information): For any discrete jointly distributed 
random vectors X n and Y n 

L{X n ;Y n ) = L x {X n -► Y n ) + L 1 (Y n ^ 1 -» X n ). (54) 
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Proof: Consider 

p(y n )p{x n 



p(y n ,x n ) 



( 6 ) nw.nN. „ p{ V n )p{x n ) 



E M^)p(2/ n )log 



p(y n \\x n )p(x n \\y n - 1 ) 



^ S p(2/"lk") mV ' & p(x"\\y^) 

= Li(x"^y n ) + Li(y"- 1 -^r), (55) 

where (a) follows from the definition of lautum information and (b) follows from the chain rule p(y n ,x n ) = 

p(tf n ||a; n )p(s n ||l/ n - 1 ). ■ 
A direct consequence of the lemma is the following condition for the equality between two types of directed 

lautum information and regular lautum information. 
Corollary 2: If 

L(X n ;Y n ) = L x (X n -4 7"), (56) 

then 

p(x n ) = p(s B ||y n-1 ) for all (x n ,y n ) e X n x y n with p{x n ,y n ) > 0. (57) 
Conversely, if d57l ) holds, then 

L(X n ; Y n ) = Li(JC" -» r) = i 2 (^" -> (58) 
Proof: The proof of the first part follows from the conservation law ( 133] ) and the nonnegativity of Kullback- 
Leibler divergence [17, Theorem 2.6.3] (i.e., Li(Y n ~ 1 — > X") = implies that p(x n ) = p(x n \\y n ~ 1 )). The second 
part follows from the definitions of regular and directed lautum information. ■ 
The lautum information rate and directed lautum information rates are respectively defined as 

C{X:Y):= lim -L(Y n ;X n ), (59) 
n — *oo 7i 

CAX -» y) := lim X") for j = 1, 2, (60) 

n— >oo 77, 

whenever the limits exist. The next lemma provides a technical condition for the existence of the limits. 

Lemma 2: If the process (X n ,Y n ) is stationary and Markov (i.e., p{xi,y,i\x l ~ 1 ^y l ~ x ) — p(xi,yi\x l ~\,y\z\) 
for some finite k), then C(X;Y) and C%{X — > Y) are well defined. Similarly, if the process {X™,Y n } ~ 
p(x n )p(y™||a;™) is stationary and Markov, then C\(X — * Y) is well defined. 

Proof: It is easy to see the sufficiency of the conditions for C\(X — > Y) from the following identity: 



ii(A-"^y i )= E p(z n )Ks/ n )iog 



p(x n )p(y n \ \x n ) 



-H(X n ) - H(Y n ) - E ^>(y")logp(x>(y"||x"). 
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Since the process is stationary the limits linin^oo ^H(X n ) and limn-^oo hH(Y n ) exist. Further- 
more, since p(x n ,y n ) = p(x n )p(y n \\x n ) is assumed to be stationary and Markov, the limit 
lim rwoo i ^2 x n yn p(x n )p(y n ) logp(x n )p(y n \ \x n ) exists. The sufficiency of the condition can be proved for 
C 2 (X — > Y) and the lautum information rate using a similar argument. ■ 
Adding causality constraints to the problems that were considered in [15], we obtain the following results for 
horse race gambling and data compression. 

A. Horse Race Gambling with Mismatched Causal Side Information 

Consider the horse race setting in Section IIII-AI where the gambler has causal side information. The joint 
distribution of horse race outcomes X n and the side information Y n is given by p(x n )p(y n \\x n ~ 1 ), namely, 
Xi — > X i_1 — > Y l form a Markov chain, and therefore the side information does not increase the growth rate. 
The gambler mistakenly assumes a joint distribution p(x n \\y n )p(y n \\x n ~ 1 ), and therefore he/she uses a gambling 
scheme b*(x n \\y n ) = p(x n \\y n ). 

Theorem 6: If the gambling scheme b*(x n \\y n ) — p(x n \\y n ) is applied to the horse race described above, 
then the penalty in the growth with respect to the gambling scheme b*(x n ) that uses no side information is 
L 2 (Y n —> X n ). For the special case where the side information is independent of the horse race outcomes, the 
penalty is L x {Y n ->• X n ). 

Proof: The optimal growth rate where the joint distribution is p(x n )p(y n | \x n ~ 1 ) is W*(X n ) — E[\ogo(X n )} — 
H{X n ). Let E p r x n\ p r y nii x n-i> denotes the expectation with respect to the joint distribution p(x n )p(y n \\x n ~ 1 ). The 
growth rate for the gambling strategy b(x n \\y n ) — p(x n \\y n ) is 

W*(X n \\Y n ) = E^wyn^-^llogbiX^Y^oiX")} 

= E^^^-^logpiX^WY")] +E p(xnMyn]lxn -^[logo(X n )}; (61) 

hence W*{X n ) — W* (X n \\Y n ) — L 2 (Y n — > X n ). In the special case, where the side information is independent 
of the horse outcome, namely, p{y n \\x n ^ 1 ) = p{y n ), then L 2 (Y n -> X n ) = L 1 (Y n ->■ X n ). ■ 
This result can be readily extended to the general stock market, for which the penalty is upper bounded by 
L 2 (Y n -^I"). 

B. Compression with Joint Distribution Mismatch 

In Section [IV] we investigated the cost of ignoring forward and backward links when compressing a jointly 
(X n ,Y n ) by an optimal lossless variable length code. Here we investigate the penalty of assuming forward and 
backward links incorrectly when neither exists. Let X n and Y n be independent sequences. Suppose we compress 
them with a scheme that would have been optimal under the incorrect assumption that the forward link p(y n \ \x n ) 
exists. The optimal lossless average variable length code under these assumptions satisfies (up to 1 bit per source 
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symbol) 

E(L(X",Y n )) = £ p(x n )p(y n )log \ - 

Jrjn p(y n \\x n )p(x n ) 

= V p(x n )p(y n ) log + H /jrnj + ff (p) 

= L 1 (X"^-F n )+lZ"(X")+iJ(F n ). (62) 

Hence the penalty is Li(X n — > Y"). Similarly, if we incorrectly assume that the backward linkp(x™||y"~ 1 ) exists, 
then 

£(l(x«,y")) = J2 p{x n )p{y n )\o g n x 

= l^y™- 1 -> x n ) + ff(x n ) + ff(y n ). (63) 

Hence the penalty is Li(Y n ^ 1 — > X"). If both links are mistakenly assumed, the penalty [15] is lautum information 
L(X n ;Y n ). Note that the penalty due to wrongly assuming both links is the sum of the penalty from wrongly 
assuming each link. This is due to the conservation law d55l l. 

C. Hypothesis Testing 

We revisit the hypothesis testing problem in Section V, which is describe in Fig. [4] As a dual to Theorem 6, we 
characterize the minimum type I error exponent given the type II error probability: 

Theorem 7 (Chernoff-Stein lemma for the causal dependence test: Type I error): Let (X, Y) = {Xi,Y}^i be 
stationary, ergodic, and Markov of finite order such that p(x n ,y n ) = implies p(x n \ \y n ~ 1 )p{y n ) — 0. Let A n C 
(X x y) n be an acceptance region, and let a n and (3 n be the corresponding probabilities of type I and type II 
errors d43j. For < e < \, let 

oSf* = min a n . (64) 

A n Q(Xxy) n J3 n <e 

Then 



lim —log oft) =C 2 {X ->y), 



(65) 

n— s-oc n 

where the directed lautum information rate is the one induced by the joint distribution from Hq, i.e., 

p(as n ||» n - 1 )p(» n ||a: w ). 

The proof of Theorem [7] follows very similar steps as in Theorem [5] upon letting 



x ,y 



log-^^-£ 2 (X-Y) 

p{y n \\x n ) K ' 



<§>, (66) 
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analogously to ( f46b , and upon using the Markov assumption for guaranteeing the AEP; hence we omit the details. 

VII. Concluding Remarks 

We have established the role of directed information in portfolio theory, data compression, and hypothesis testing. 
Put together with its key role in communications [7]-[14], and in estimation [21], directed information is thus 
emerging as a key information theoretic entity in scenarios where causality and the arrow of time are crucial to the 
way a system operates. Among other things, these findings suggest that the estimation of directed information can 
be an effective diagnostic tool for inferring causal relationships and related properties in a wide array of problems. 
This direction is under current investigation [22]. 
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